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(54) Robot and action deciding method lor robot 

(57) A robot device I has a sensor 1 01 for detecting 
information of a user, a user identification section 120 
for identifying one user from a plurality of identifiable us- 
ers on the basis of the information of the user detected 



by the sensor 101 , and an action schedule section 130, 
an action instruction execution section 1 03 and an out- 
put section 104 as action control means for manifesting 
an action corresponding to the one user identified by the 
user identification section 120. 
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Description 

El ™ S invention rela,es t0 a robot and an action dmMno method for deciding the action of the robot 
0002] Recently, there has been proposed a robot which autonomously acts in accordance with ambient information 
(external elements) and internal information (internal elements). For example, such a robot is exemplified by a so- 
called pet robot as a robot device in the format of an animal, a mimic organism, or a virtual organism displayed on a 
display or the like of a computer system. 

[0003] The above-described robot devices can autonomously act, for example, in accordance with a word or an 
instruction from a user. For example, the Japanese Publication of Unexamined Patent Application No.H1 0-289006 
discloses a technique of deciding the action on the basis of pseudo emotions. 

[0004] Meanwhile, all the conventional robot devices react in the same manner to every user. That is the robot 
devices react uniformly to different users and do not change their reactions depending on the users 
[0005] If the robot devices identify the users and react differently to the different users, it is possible to enjoy inter- 
actions with each user ' 

[0006] Thus, in view of the foregoing status of the art, it is an object of the present invention to provide a robot which 
reacts differently to different users, and an action deciding method for the robot. 
[0007] A robot according to the present invention comprises: detection means for detecting information of a user 
identification means for identifying one user from a plurality of identifiable users on the basis of the information of the 
user detected by the detection means; and action control means for manifesting an action corresponding to the one 
user identified by the identification means. 

[0008] In the robot having such a structure, one user is identified from a plurality of identifiable users by the identi- 
ication means on thebasis of the information of the user detected by the detection means, and an action corresponding 
to the one user identified by the identification means is manifested by the action control means 
[0009] Thus, the robot identifies one user from a plurality of identifiable users and reacts corresponding to the one 
* 5 user. 

[0010] An action deciding method for a robot according to the present invention comprises the steps of identifvina 
one user from a plurality of identifiable users on the basis of information of the user detected by detection means and 
manifesting an action corresponding to the identified one user. 

«. J" accordance with this action deciding method for a robot, the robot identifies one user from a plurality of 

30 identifiable users and reacts corresponding to the one user. 

[0012] An embodiment of the invention will now be described, by way of example only, with reference to the accom- 
panying drawings in which: 

Fig. 1 is a perspective view showing the exterior structure of a robot device as an example embodiment of the 
•» present invention. 

Fig.2 is a block diagram showing the circuit structure of the robot device. 
Fig.3 is a block diagram showing the software configuration ofthe robot device. 

Fig 4 is a block diagram showing the configuration of a middleware layer in the software configuration of the robot 
device. 

Rg.5 is a block diagram showing the configuration of an application layer in the software configuration of the robot 
device. 

Fig.6 is a block diagram showing the configuration of an action model library in the application layer 
Fig.7 is a view for explaining a finite probability automaton, which is information for action decision of the robot 
device. 

Fig.8 shows a state transition table prepared for each node of the finite probability automaton. 
Fig.9 is a block diagram showing a user recognition system of the robot device. 

Fig.1 0 is a block diagram showing a user identification section and an action schedule section in the user recoq- 
nition system. a 

Fig.1 1 is a block diagram showing a user registration section in the user recognition system 
F.g.1 2 shows action schedule data as action information of the robot device, in which a finite probability automaton 
corresponding to a plurality of users is used. 

Fig.13 shows action schedule data as action information of the robot device, in which a part of a finite probability 
automaton is prepared in accordance with a plurality of users. 

55 F V A S , h °^ th ! 0386 Where transltion P roba °i'ity data of a finite probability automaton is prepared in accordance 

DD with a plurality of users. 

Fig 15 is a block diagram showing the specific structure of the user identification section in the user recognition 
system. 

Fig.1 6 is a graph for explaining a registered contact pattern. 
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Fig.1 7 is a graph for explaining an actually measured contact pattern. 

Fig.1 8 is a graph for explaining dispersion of evaluation information of the user. 

Fig. 1 9 is a flowchart showing the procedure for obtaining an actually measured contact pattern and obtaining an 
evaluation signal. 

5 

[001 3] A preferred example embodiment of the present invention will now be described herein in detail with reference 
to the accompanying drawings. In this embodiment, the present invention is applied to a robot device which autono- 
mously acts in accordance with ambient information and internal information (information of the robot device itself). 
[0014] In the embodiment, the structure of the robot device will be described first, and then the application of the 
10 present invention to the robot device will be described in detail. 

(1) Structure of Robot Device According to Embodiment 

[0015] As shown in Fig.1 , a robot device 1 is a so-called pet robot imitating a "dog". The robot device 1 is constituted 
15 by connecting limb units 3A, 3B, 3C and 3D to front and rear portions on the right and left sides of a trunk unit 2 and 
connecting a head unit 4 and a tail unit 5 to a front end portion and a rear end portion of the trunk unit 2, respectively. 
[0016] In the trunk unit 2, a control section 1 6 formed by interconnecting a CPU (central processing unit) 1 0, a DRAM 
(dynamic random access memory) 11 , a flash ROM (read only memory) 12, a PC (personal computer) card interface 
circuit 13 and a signal processing circuit 14 via an internal bus 15, and a battery 17 as a power source of the robot 
20 device 1 are housed, as shown in Fig.2. Also, an angular velocity sensor 18 and an acceleration sensor 1 9 for detecting 
the direction and acceleration of motion of the robot device 1 are housed in the trunk unit 2. 

[0017] In the head unit 4, a CCD (charge coupled device) camera 20 for imaging the external status, a touch sensor 

21 for detecting the pressure applied through a physical action like "stroking" or "hitting" by a user, a distance sensor 

22 for measuring the distance to an object located forward, a microphone 23 for collecting external sounds, a speaker 
25 24 for outputting a sound such as a bark, and a LED (light emitting diode) (not shown) equivalent to the "eyes" of the 

robot device 1 are arranged at predetermined positions. 

[0018] Moreover, at the joint portions of the limb units 3A to 3D, the connecting portions between the limb units 3A 
to 3D and the trunk unit 2, the connecting portion between the head unit 4 and the trunk unit 2, and the connecting 
portion of a tail 5A of the tail unit 5, actuators 2S A to 25 n and potentiometers 26 1 to 26 n having corresponding degrees 

30 of freedom are provided. 

[0019] These various sensors such as the angular velocity sensor 1 8, the acceleration sensor 1 9, the touch sensor 
21 , the distance sensor 22, the microphone 23, the speaker 24 and the potentiometers 26! to 26 n , and the actuators 
25., to 25 n are connected with the signal processing circuit 1 4 of the control section 1 6 via corresponding hubs 27.| to 
27 n . The CCD camera 20 and the battery 17 are directly connected with the signal processing circuit 14. 

35 [0020] The signal processing circuit 1 4 sequentially takes therein sensor data, image data and sound data supplied 
from the above-described sensors, and sequentially stores these data at predetermined positions in the DRAM 11 via 
the internal bus 15. Also, the signal processing circuit 14 sequentially takes therein remaining battery capacity data 
expressing the remaining battery capacity supplied from the battery 1 7 and stores this data at a predetermined position 
in the DRAM 11. 

40 [0021] The sensor data, image data, sound data, and remaining battery capacity data thus stored in the DRAM 11 
are later used by the CPU 10 for controlling the operation of the robot device 1 . 

[0022] In practice, in the initial state when the power of the robot device 1 is turned on, the CPU 10 reads out, directly 
or via the interface circuit 13, a control program stored in a memory card 28 charged in a PC card slot, not shown, in 
the trunk unit 2 or stored in the flash ROM 12, and stores the control program into the DRAM 11 . 
45 [0023] Later, the CPU 10 discriminates the status of the robot device itself, the ambient status, and the presence/ 
absence of an instruction or action from the user, on the basis of the sensor data, image data, sound data, remaining 
battery capacity data which are sequentially stored into the DRAM 1 1 from the signal processing circuit 1 4 as described 
above. 

[0024] Moreover, the CPU 1 0 decides a subsequent action on the basis of the result of discrimination and the control 
so program stored in the DRAM 11 , and drives the necessary actuators 25^ to 25 n on the basis of the result of decision. 
Thus, the CPU 1 0 causes the robot device 1 to shake the head unit 4 up/down and left/right, or to move the tail 5A of 
the tail unit 5, or to drive the limb units 3A to 3D to walk. 

[0025] In this case, the CPU 1 0 generates sound data, if necessary, and provides this sound data as a sound signal 
via the signal processing circuit 1 4 to the speaker 24, thus outputting a sound based on the sound signal to the outside. 
55 The CPU 1 0 also turns on or off the LED, or flashes the LED. 

[0026] In this manner, the robot device 1 can autonomously act in accordance with the status of itself, the ambient 
status, and an instruction or action from the user. 
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(2) Software Configuration of Control Program 

aTHi !; 0f,War ! COn, i 9lJra,ion of the above-described control program in the robot device 1 is as shown in Fig. 
3. In F.g.3, a dev.ce dnver layer 30 is located on the lowermost layer of the control program and is constituted by a 

to directly access the hardware used in an ordinal computer such as the CCD camera 20 (Fig 2) and a timer and 
carries out process.ng in response to an interruption from the corresponding hardware 

SSSL ,1^'° T 6 ' °^ jeCt l 2 iS '° Cated ° n an Upper ,ayer than the device driver 30, and is constituted by 

LShLf ot33madeupofa * 0 ^ aregr ^ 

dlifJL TT S0UrC !' 8 d6ViCe d ™ 6r mana9er 35 made Up 0f a software 9™P ,or managing various ofher 
dev,cednvers,andades,gned robot 36 madeupofasoftwaregroupformanagingthemechanismoftherobotdevicel 
obZrlntlTn- J6 1 37 iS COns,ituted bv an a** manager 38 and a service manager 39. In this case, the 
object manager 38 is a software group for managing the start-up and termination of the software groups contained in 
the robotic server ob ect 32, a middleware layer 40 and an application layer 41. The service manager 39 is a software 
JrlTT 9 L connectl0n of on tne ba sis of connection information between objects described in a 

connection file stored in the memory card 28 (Fig.2). 

SIS middleware J aver 40 is located °n a " layer than the robotic server object 32 and is constituted by 

IuoSCh" T : S '° C f ^ 0n a " UPP6r ,ay6r th3n ,h6 middleWar6 laver 40 and is cons tituted by a software 
group for deeding the action of the robot device 1 on the baste of the result of processing earned out by ihe software 
group constituting the middleware layer 40. y sonware 

Figs^andS 6 r^SeC^ C ° nfi9Urati ° nS ° f the midd,eware lave ' 4 ° «** the application layer 41 are shown in 

" { S%)J^hTT Wa - rS ,ay6r 40 iS COnstituted by: a cognition system 60 having signal processing modules 50 to 
ti ion Zt ' te 7 erature detection ' bri 9 htness lection, scale recognition, distance detection, posture de- 
STnn'it S ! t «o ^ eteCti ° n ' and C ° ,0r rec °9 nition ' and a " ^put semantics converter module 59; and a 
recognition system 69 having an output semantics converter module 68, and signal processing modules 61 to 67 for 

- sn s a r 0 r;%T n9 ' motion reproduction - waikin9, restoration fram tumb,e led *** and s °- d -p'° 

£nfn! h2 eSi9nal H ? CeSS J n9 m ° dUleS 50 10 58 in the rec °3 nition ^tem 60 take therein suitable data of the various 
obe?^ 

P T predetem,ned P rocessin 9 ba sed on the data, and provides the result of processing to the 
input semant.es converter module 59. In this case, the virtual robot 33 is constituted as a unit for suppMng/raeiv no 
35 or converting signals in accordance with a predetermined protocol. supplying/receiving 

l<Xm } J h VT! Semantics conv eter module 59 recognizes the status of itself and the ambient status such as "it is 

Tm ' s i' S » h i V? ri9htVI de,eCtSd 3 ba "" detected a tumb,e " was "I was hit", "I hea" a iale o 

do-m -sol , I detected a moving object", or "I detected an obstacle", and an instruction or action from the and 
outputs the result of recognition to the application layer 41 (Fig 5) ' 

tT^ f PP ' iCati0n laV ? r 41 is con stituted by five modules, that is, an action model library 70, an action switching 
mnt ' , h 9 m ° dUle 72 ' a " em ° ti0n m ° del 73 ' and an instinct ™ d e' 74, as shown in Fig 5 9 
the "J. 0 " model , librar y 70 - ^dependent action models 70, to 70 n are provided corresponding to several 
cond.tion rtems which are selected in advance such as "the case where the remaining battery capacity is short" "he 
case of restoring from a tumble", "the case of avoiding an obstacle", "the case of expressing an emotion" and " he 
case where a ball is detected", as shown in Fig.6. emouon , ana me 

StiJ!^ T* recognition is provided fr °m the input semantics converter module 59 or when a predeter- 
mine 7n PaSS6d S rV he laSt reC ° 9niti0n reSUft Wa * provided ' the action models 70, to 70 n decide subsequent 
and ^ Da rlT Sary ' " referenCG 10 3 Va ' Ue ° f 3 pending emotion held in the emotion model 73 

,h r J^Z h A" COrreSpondir, 9 desire held in the instinct model 74 as will be described .a.er, and output 
the results of decision to the action switching module 71 . v 

^IJl^LT^TiT 8 teChniqUe ° f deC,din9 s" b sequent actions, the action models 70, to 70 n use an 
algorrthm ca ed f.nrte probability automaton such that which one of nodes (states) NOOE 0 to NODE becomes tSe 

S^J^TrS^ZT °' n ° d6S NODEot ° N ° DE " fe dedded!n ternis°of probablty on the biis 
* snownt Fig 7 1 " S ^ AR ° 1 * ARC " 1 C ° nneCting the respective nodes NODE o *> NODE n , as 

Snnrf tnnc e aCti ° n m ° delS 7 ° 1 *° 7 °" have 3 state transiti on table 80 as shown in Fig.8 for each of the 
node NODEo to NODE n , corresponding to the nodes NODE 0 to NODE,, forming their respective action models 70, to 
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[0040] In the state transition table 80, input events (results of recognition) as transition conditions at the nodes NODE 0 
to NODE n are listed in the row of "name of input event" In the preferential order, and further conditions with respect to 
the transition conditions are described the corresponding columns in the rows of "name of data" and "range of data". 
[0041] Therefore, at a node NODE 100 shown in the state transition table 80 of Flg.8, the conditions for transition to 
5 another node are that if the result of recognition to the effect that "a ball is detected (BALL)" is provided, the "size 
(SIZE)" of the ball provided together with the result of recognition is within a range of "0 to 1 000", and that if the result 
of recognition to the effect that "an obstacle is detected (OBSTACLE)" is provided, the "distance (DISTANCE)" to the 
obstacle provided together with the result of recognition is within a range of "0 to 1 00". 

[0042] At this node NODE 100 , even if there is no input of any result of recognition, transition to another node can be 
10 made when the parameter value of any of "joy", "surprise" and "sadness" held in the emotion model 73 Is within a 
range of "50 to 100", of the parameter values of emotions and desires held in the emotion model 73 and the Instinct 
model 74 which are periodically referred to by the action models 70 1 to 70 n . 

[0043] In the state transition table 80, the name of nodes to which transition can be made from the nodes NODE 0 
to NODE n are listed in the column of "transition destination node" in the section of "transition probability to other nodes". 

15 Also, the transition probabilities to the other nodes NODE 0 to NODE n to which transition can be made when all the 
• conditions described in the rows of "name of input event", "name of data" and "range of data" are met are described 
in corresponding parts in the section of "transition probability to other nodes". Actions that should be outputted in 
transition to the nodes NODE 0 to NODE n are described in the row of "output action" in the section of "transition prob- 
ability to other nodes". The sum of the probabilities of the respective rows in the section of "transition probability to 

20 other nodes" is 1 00 [%]. 

[0044] Therefore, at the node NODE 100 shown in the state transition table 80 of Fig.8, for example, if there is provided 
the result of recognition to the effect that "a ball is detected (BALL)" and that the "size" of the ball is within a range of 
"0 to 1000", transition to a "node NODE 120 " can be made with a probability of "30 [%]" and an action of "ACTION 1" is 
outputted then. 

25 [0045] The actions models 70-, to 70 n are constituted so that a number of such nodes NODE 0 to NODE n described 
in the form of the state transition tables 80 are connected. When the result of recognition is provided from the input 
semantics converter module 59, the actions models 70, to 70 n decide next actions in terms of probability by using the 
state transition tables of the corresponding nodes NODE 0 to NODE n and output the results of decision to the action 
switching module 71. 

30 [0046] In a user recognition system, which will be described later, different action models for constructing action 
information based on the finite probability automaton are provided for different users, and the robot device 1 decides 
its action in accordance with the action model (finite probability automaton) corresponding to the identified one user. 
By changing the transition probability between nodes, the action is varied for each identified user. 
[0047] The action switching module 71 selects an action outputted from the action model of the action models 70 1 

35 to 70 n that has the highest predetermined priority, of the actions outputted from the action models 70-, to 70 n of the 
action model library 70, and transmits a command to the effect that the selected action should be executed (hereinafter 
referred to as action command) to the output semantics converter module 68 of the middleware layer 40. In this em- 
bodiment, higher priority is set for the action models 70 1 to 70 n described on the lower side in Fig.6. 
[0048] On the basis of action completion information provided from the output semantics converter module 68 after 

40 the completion of the action, the action switching module 71 notifies the learning module 72, the emotion model 73 
and the instinct model 74 of the completion of the action. 

[0049] The learning module 72 inputs the result of recognition of teaching received as an action from the user, like 
"being hit" or "being stroked", of the results of recognition provided from the input semantics converter module 59. 
[0050] On the basis of the result of recognition and the notification from the action switching module 71 , the learning 
45 module 72 changes the transition probabilities corresponding to the action models 70 1 to 70 n in the action model library 
70 so as to lower the probability of manifestation of the action when it is "hit (scolded)" and to raise the probability of 
manifestation of the action when it is "stroked (praised)". 

[0051 ] The emotion model 73 holds parameters indicating the strengths of 6 emotions in total, that is, "joy", "sadness", 
"anger", "surprise", "disgust", and "fear". The emotion model 73 periodically updates the parameter values of these 

so emotions on the basis of the specific results of recognition such as "being hit" and "being stroked" provided from the 
input semantics converter module 59, the lapse of time, and the notification from the action switching module 71 . 
[0052] Specifically, the emotion model 73 calculates a parameter value E[t+ 1] of the emotion in the next cycle, using 
the following equation (1 ), wherein AE[t] represents the quantity of variance in the emotion at that time point calculated 
in accordance with a predetermined operation expression on the basis of the result of recognition provided from the 

55 input semantics converter module 59, the action of the robot device 1 at that time point and the lapse of time from the 
previous update, and ^ represents a coefficient indicating the intensity of the emotion. The emotion model 73 then 
updates the parameter value of the emotion by replacing it with the current parameter value E[t] of the emotion. The 
emotion model 73 similarly updates the parameter values of all the emotions. 
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E[t+1] = E[tJ + kex AE[t] 



d) 



[0053] To what extent the results of recognition and the notification from the output semantics converter module 68 
influence the quantity of variance AE[t] in the parameter value of each emotion to predetermined Fo exlple tne 

the emotion of "anger", and the result of recognition to the effect that it was "stroked" la gely affects the ouan S^o 
vanance AE[t] in the parameter value of the emotion of "joy". q ^ 

[0054J The notification from the output semantics converter module 68 is so-called feedback information of the action 
(act on completion information), that is, information about the result of manifestation of the action ™ e emtt to ™del 
73 also changes ; the emotions in accordance wfth su^ 

leaSodutS 1 tf f 9 "- V n °!i fi ? ati ° n fTOm the 0UtpUt Semantics ~ er ™ dule 68 - alsoCed ^ he 

Se 7?(:«rtro;r H of the action may ateo be carried out ,hrou9h the output ° f the action switch ^ 

EL 2 fir!!"* T dSl 74 h0 ' d , S parameters indicatin 9 tne snaths of 4 desires which are independent of one 

from thai nT d,Ca,ly t Updat6S ,he Peeler values of these desires on the basis of the results of recogTftfon ^ Wed 
from the input semantics converter module 59, the lapse of time, and the notification from the action switching module 

[0057] Specifics, with respect to "desire for exercise", "desire for affection" and "curiosity" the instinct model 74 

c2e whlTrr Va,US ' tk+1] ° f thS deSire in the neXt CyCle ' USi "9 the followi "9 "W*** 2) I a £££££ 
cycle, wherein Al[k] represents the quantity of variance in the desire at that time point calculated in accordance w "h a 

S^rV*^" 6XPreSSi0n ° n ,h6 baSiS °' ,hC reSUltS 0f the la P*e °' ™£EZSS!£ 

SlSSSSST C °T er r'" 16 68> ^ * repr6SentS a Coefficient indicatin ° *• 'tensity of the 5™ 
The instinct model 74 then updates the parameter value of the desire by replacing the result of calculation with he 

^X^T' m 01 the desire - The instinct model 74 simi,arty updates ,he v2?JS,2L2 



l[k+1] = l[k] + ki x Al[k] (2) 

[0058] To what extent the results of recognition and the notification from the output semantics converter module 68 
T"*? °! VarianCS m " the Parameter Va ' Ue ° f each deslre is P^eteZed F Tex^ p rrno 
SorSgue" ° " Seman, ' CS C ° nVerter m ° dUle 68 lar96ly aff6Cte the qUantity ° f VaHanCe A 'M the'paleter 
[0059] The parameter value may also be decided in the following manner. 

El n r t e T Ple ' f parameter value of 'P ain ° is Prided. "Pain" affects "sadness" in the emotion model 73 
[0061] On thebas,s of the number of times an abnonnal posture is taken, notified of via the signal process!! module 

hi Unl IS # CalCU,ated " S,nd the f ° ,lowil1 9 e °^ion O). wherein N represents the number of times Zep eJeZ 
he strength of pain, and K 2 represents a constant of the speed of reduction in pain. Then the p^^ZTS 



l[k] = K, x N - K, x t 



(3) 



oTfLn TT ^ f Parameter Va ' Ue ° f " ,ever is provided - 0n the basis °' temperature data from the signal 
ESSiTT* 1 ?- 61 f ? f tem P erature detec «°". Prided via the input semantics converter module 59 a paramele 
. ,S f a ' CUlated usin 9 the followina (4). wherein T represents the tenjeratu^ t!£S2I 

uod^H h„ tenlperatu : e ' 3nd Ka repr6SentS 3 tem P^ture rise coefficient. Then, the parameter value , 0 W "J 
u dated by replacing the result of calculation w*h the current parameter va.ue l[k] of fever. If T-T 0 is less than 0 JM = 
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1[k] = (T-T 0 )xK 3 



(4) 



[0063] With respect to "appetite 0 in the instinct model 74, on the basis of the remaining battery capacity data (infor- 
mation obtained by a module for detecting the remaining battery capacity, not shown) provided via the input semantics 
converter module 59, a parameter value l[k] of "appetite" is calculated using the following equation (5) in a predeter- 
mined cycle, wherein B L represents the remaining battery capacity. Then, the parameter value of "appetite" is updated 
by replacing the result of calculation with the current parameter value l[k] of appetite. 



[0064] Alternatively, a parameter value of "thirst" is provided. On the basis of the speed of change in the remaining 
battery capacity provided via the input semantics converter module 59, a parameter value l[k] of "thirst" is calculated 
using the following equation (6) wherein B L (t) represents the remaining battery capacity at a time point t and the re- 
maining battery capacity data is obtained at time points t 1 and t 2 . Then, the parameter value of "thirst" is updated by 
replacing the result of calculation with the current parameter value l[k] of thirst. 



[0065] In the present embodiment, the parameter values of the emotions and desires (instincts) are regulated to vary 
within a range of 0 to 100. The values of the coefficients and k, are individually set for each of the emotions and 
desires. 

[0066] Meanwhile, the output semantics converter module 68 of the middleware layer 40 provides abstract action 
commands such as "move forward", "be pleased", "bark or yap", or "tracking (chase a ball)" provided from the action 
switching module 71 in the application layer 41 , to the corresponding signal processing modules 61 to 67 in the rec- 
ognition system 69, as shown in Fig.4. 

[0067] As the action commands are provided, the signal processing modules 61 to 67 generate servo command 
values to be provided to the corresponding actuators 25^ to 25 n for carrying out the actions, and sound data of a sound 
to be outputted from the speaker 24 (Fig.2) and/or driving data to be supplied to the LED of the "eyes", on the basis 
of the action commands. The signal processing modules 61 to 67 then sequentially transmit these data to the corre- 
sponding actuators 25 1 to 25 n , the speaker 24, or the LED, via the virtual robot 33 of the robotic server object 32 and 
the signal processing circuit 14 (Fig.2). 

[0068] In this manner, on the basis of the control program, the robot device 1 can autonomously acts in response to 
the status ofthe device itself, the ambient status, and the instruction or action from the user. 

(3) Change of Instinct and Emotion in Accordance with Environment 

[0069] In the robot device 1 , in addition to the above-described configuration, the emotions and instincts are changed 
in accordance with the degrees of three conditions, that is, "noise", "temperature", and "illuminance" (hereinafter re- 
ferred to as ambient conditions), of the ambient. For example, the robot device 1 becomes cheerful when the ambient 
is "bright", whereas the robot device 1 becomes quiet when the ambient is "dark". 

[0070] Specifically, in the robot device 1 , a temperature sensor (not shown) for detecting the ambient temperature 
is provided at a predetermined position in addition to the CCD camera 20, the distance sensor 22, the touch sensor 
21 and the microphone 23 as the external sensors for detecting the ambient status. As the corresponding configuration, 
the signal processing modules 50 to 52 for noise detection, temperature detection, and brightness detection are pro- 
vided in the recognition system 60 ofthe middleware layer 40. 

[0071 ] The signal processing module for noise detection 50 detects the ambient noise level on the basis of the sound 
data from the microphone 23 (Fig.2) provided via the. virtual robot 33 of the robotic server object 33, and outputs the 
result of detection to the input semantics converter module 59. 

[0072] The signal processing module for temperature detection 51 detects the ambient temperature on the basis of 
the sensor data from the temperature sensor provide via the virtual robot 33, and outputs the result of detection to the 
input semantics converter module 59. 

[0073] The signal processing module for brightness detection 52 detects the ambient illuminance on the basis ofthe 
image data from the CCD camera 20 (Fig.2) provided via the virtual robot 33, and outputs the result of detection to the 
input semantics converter module 59. 



I[k]= 100 -B L 



(5) 



l[k] = {B L (t 2 )-B L (t 1 ))/(t 2 -t 1 ) 



(6) 
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ognition to the internal state models In the application layer 41 (Fig 5) 
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[0089] The robot device 1 , constituted as described above, can change the state of emotions and instincts in ac- 
cordance with the information of the robot device itself and the external information, and can autonomously act in 
response to the state of emotions and instincts. 

5 (4) Structure for User Recognition 

[0090] The application of the present invention to the robot device will now be described in detail. The robot device 
to which the present invention is applied is constituted to be capable of identifying a plurality of users and reacting 
differently to the respective users. A user identification system of the robot device 1 which enables different reactions 
10 to the respective users is constituted as shown in Fig.9. 

[0091] The user identification system has a sensor 1 01 , a user registration section 11 0, a user identification section 
120, a user identification information database 102, an action schedule section 130, an action instruction execution 
section 103, and an output section 104. 

[0092] In the user identification system, the user identification section 120 identifies users on the basis of an output 
15 from the sensor 1 01 . In this case, one user is identified with reference to information about a plurality of users which 
is registered in advance in the user identification information database 102 by the user registration section 110. The 
action schedule section 130 generates an action schedule corresponding to the one user on the basis of the result of 
identification from the user identification section 120, and an action is actually outputted by the action instruction ex- 
ecution section 1 03 and the output section 1 04 in accordance with the action schedule generated by the action schedule 
20 section 130. 

[0093] In such a structure, the sensor 101 constitutes detection means for detecting information about a user, and 
the user identification section 1 20 constitutes identification means for identifying one user from a plurality of identifiable 
users on the basis of the information about a user detected by the sensor 101 . The action schedule section 130, the 
action instruction execution section 103 and the output section 104 constitute action control means for causing man- 
25 ifestation of an action corresponding to the one user identified by the user identification section 120. 

[0094] The user registration section 110 constitutes registration means for registering information about a plurality 
of users (user identification information) to the user identification information database 1 02 in advance. The constituent 
parts of such a user identification system will now be described in detail. 

[0095] The user identification section 120 identifies one user from a plurality of registered users. Specifically, the 
30 user identification section 120 has a user information detector 121 , a user information extractor 122 and a user iden- 
tification unit 123, as shown in Fig.10, and thus identifies one user. 

[0096] The user information detector 121 converts a sensor signal from the sensor 1 01 to user identification infor- 
mation (user identification signal) to be used for user identification. The user information detector 121 detects the 
characteristic quantity of the user from the sensor signal and converts it to user identification information. In this case, 
35 the sensor 101 may be detection means capable of detecting the characteristics of the user, like the CCD camera 20 
shown in Fig.2 for detecting image information, the touch sensor 21 for detecting pressure information, or the micro- 
phone 23 for detecting sound information. For example, the CCD camera 20 detects a characteristic part of the face 
as the characteristic quantity, and the microphone 23 detects a characteristic part of the voice as the characteristic 
quantity. 

*o [0097] The user information detector 1 21 outputs the detected user identification information to the user identification 
unit 1 23. Information from the user information extractor 1 22 (registered user identification information) is also inputted 
to the user identification unit 123. 

[0098] The user information extractor 1 22 extracts the user identification information (user identification signal) which 
is registered in advance, from the user identification information database 102 and outputs the extracted user identi- 
45 fjcation information (hereinafter referred to as registered user identification information) to the user identification unit 
123. 

[0099] In this case, the user identification information database 1 02 is constructed by a variety of information related 
to users, including the registered user identification information for user identification. For example, the characteristic 
quantity ofthe user is used as the registered user identification information. Registration of the user identification in- 
so formation to the user identification information database 1 02 is carried out by the user registration section 1 1 0 shown 
in Fig.9. 

[0100] Specifically, the user registration section 110 has a user information detector 111 and a user information 
registerer 112, as shown in Fig. 11 . 

[01 01 ] The user information detector 1 1 1 detects information (sensor signal) from the sensor 1 01 as user identification 
55 information (user identification signal). In the case where the sensor 101 is the CCD camera 20, the touch sensor 21 
or the microphone 23 as described above, the user information detector 111 outputs image information, pressure in- 
formation or sound information outputted from such a sensor 1 01 to the user information registerer 112 as user iden- 
tification information. 
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[0117] The action schedule selector 131 holds a plurality of finite probability automatons (action schedule data) DT1 , 
DT2, DT3, DT4 corresponding to a plurality of users, as shown In Flg.1 2. Thus, the action schedule selector 131 selects 
a corresponding finite probability automaton in accordance with the user label and outputs the selected finite probability 
automaton to the action instruction selector 132. The action instruction selector 132 outputs an action instruction signal 

5 on the basis of the finite probability automaton selected by the action schedule selector 131 . 

[0118] Alternatively, the action schedule selector 131 can hold finite probability automatons for prescribing actions, 
with a part thereof corresponding to each user, as shown in Fig.13. That is, the action schedule selector 131 can hold 
a finite probability automaton DM of a basic part and finite probability automatons DS1 , DS2, DS3, DS4for respective 
users, as the action schedule data. 

10 [0119] In the example shown in Fig.12, one finite probability automaton is held as complete data corresponding to 
a plurality of users. However, as shown in Fig.13, it is also possible to hold a part of the finite probability automaton 
for each user. Although the feature of the present invention is that the robot device 1 reacts differently to different users, 
the reaction need not necessarily be different with respect to all the actions and some of general actions may be 
common. 

15 [0120] Thus, the action schedule selector 131 holds a part of the finite probability automaton in accordance with a 
plurality of users. In such a case, by setting a basic node in the finite probability automaton DM of the basic part and 
the finite probability automatons DS1 , DS2, DS3, DS4 prepared specifically for the respective users, It is possible to 
connect two finite automatons and handle them as a single piece of information for action decision. 
[0121] By thus holding a part of the finite probability automaton in accordance with a plurality of users instead of 

20 holding the entire finite probability automaton, the quantity of data to be held can be reduced. As a result, the memory 
resource can be effectively used. 

[0122] The action schedule selector 1 31 can also hold action schedule data corresponding to each user as transition 
probability data DP, as shown in Fig.14. 

[0123] As described above, the finite probability automaton prescribes transition between nodes by using the prob- 
es ability. The transition probability data can be held in accordance with a plurality of users. For example, as shown in 
Fig.14, the transition probability data DP is held corresponding to a plurality of users in accordance with the address 
of each arc in the finite probability automaton DT. In the example shown in Fig.14, the transition probability data of arcs 
connected from nodes a A", "B", n C", ... to other nodes are held and the transition probability of the arc of the finite 
probability automaton is prescribed by the transition probability data of "user 2". 
30 [0124] As the transition probability provided for the arc of the finite probability automaton is held for each user, it is 
possible to prepare uniform nodes (postures or motions) regardless of the user and to vary the transition probability 
between nodes depending on the user. Thus, the memory resource can be effectively used in comparison with the 
case where the finite probability automaton is held for each user as described above. 

[0125] The action schedule data as described above is selected by the action schedule selector 131 in accordance 
35 with the user, and the action instruction selector 132 outputs action instruction information on the basis of the action 
schedule data to the action instruction execution section 1 03 on the subsequent stage. 

[0126] The action instruction execution section 103 outputs a motion instruction signal for executing the action on 
the basis of the action instruction signal outputted from the action schedule section 130. Specifically, the above-de- 
scribed output semantics converter module 68 and the signal processing modules 61 to 67 correspond to these sec- 
40 tions. 

[0127] The output section 104 is a moving section driven by a motor or the like in the robot device 1 , and operates 
on the basis of the motion instruction signal from the action instruction execution section 103. Specifically, the output 
section 104 is each of the devices controlled by commands from the signal processing module 61 to 67. 
[0128] The structure of the user identification system and the processing in each constituent section are described 

45 above. The robot device 1 identifies the user by using such a user identification system, then selects action schedule 
data corresponding to the user on the basis of the result of identification, and manifests an action on the basis of the 
selected action schedule data. Thus, the robot device 1 reacts differently to different users. Therefore, reactions based 
on interactions with each user can be enjoyed and the entertainment property of the robot device 1 is improved. 
[0129] In the above-described embodiment, the present invention is applied to the robot device 1 . However, the 

50 present invention is not limited to the this embodiment. For example, the user identification system can also be applied 
to a mimic organism or a virtual organism displayed on a display of a computer system. 

[0130] In the above-described embodiment, the action schedule data prepared for each user is a finite probability 
automaton. However, the present invention is not limited to this. What is important is that data such as an action model 
for prescribing the action of the robot device 1 is prepared for each user. 
55 [0131] It is also possible to prepare a matching set for each user. A matching set is an information group including 
a plurality of pieces of information for one user. Specifically, the information group includes characteristic information 
for each user such as different facial expressions and different voices obtained with respect to one user. 
[0132] After specifying (identifying) the user, pattern matching of a facial expression or an instruction from the user 
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[0146] ,n the case where the data of the registered contact pattern is expressed by Di[ti,p] (i is an integer), where t 
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represents a dimensionless quantity of time (time element) and p represents an output value of the pressure sensor 
(detection signal element), the registered contact pattern shown in Fig. 16 includes a set D of five data (i = 1 , 2, .... 5), 
that is, contact data Dl of the pressure sensor 1 01 on the chin portion, non-contact data D2 of the pressure sensors, 
first contact data D3 of the pressure sensor 1 01a2 on the head portion, non-contact data D4 of the pressure sensors, 
s and second contact data D5 of the pressure sensor 1 01 on the head portion, as shown in the following Table 1 . 



Table 1 



Dl 


-[tl.p2] 


= [0.25, 2] 


D2 


= [t2, 0] = 


= [0.125,0] 


D3 


-[t3.pl] 


= [0.25, 1] 


D4 


- [t4, 0] = 


= [0.125,0] 


D5 


-[t5.pl] 


= [0.25, 1] 



25 [0147] The dimensionless quantity of time is made dimensionless on the basis of the total time T 
(1 00+50+1 00+50+1 00 [msec]) of the registered contact pattern. p1 is an output value (for example, "1 ") of the pressure 
sensor 101a! on the chin portion, and p2 is an output value (for example, "2") of the pressure sensor 101a 2 on the 
head portion. The purpose of using the dimensionless time as the data of the contact pattern is to eliminate the time 
dependency and realize robustness in the conversion to the evaluation signal by the stroking manner evaluation signal 

30 calculation section 143. 

[01 48] A user who intends to be identified through comparison with the registered contact pattern as described above 
needs to stroke the pressure sensor 101a in such a manner as to match the registered pattern. For example, it is 
assumed that an actually measured contact pattern as shown in Fig.17 is obtained as the user operates the pressure 
sensors 101a 1t 101a 2 on the chin portion and the head portion in trying to be identified. 

35 [0149] If the data of the actually measured contact pattern is expressed by Di'[ti\p] (i is an integer), where t' represents 
a dimensionless quantity of time, the actually measured contact pattern shown in Fig.17 includes a set D' (= D1\ D2\ 
D3', D4\ D5') of five data (i = 1 , 2, .., 5) D1\ D2\ D3\ D4\ D5' as shown in the following Table 2. 



Table 2 



Dl' = [tl\p2] = [0.275,2] 
45 D2 , = [t2\0] = [0.15,0] 

D3' = [t3',pl] = [0.3, 1] 
50 D4' = [t4\0] = [0.075,0] 

D5' = [t5», pi] = [0,2,1] ; 

55 

[0150] The stroking manner evaluation signal calculation section 1 43 compares the actually measured contact pat- 
tern expressed in the above-described format, with the registered contact pattern. At the time of comparison, the reg- 
istered contact pattern is read out from the user identification information database 1 02 by the user information extractor 
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122. 
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Su is provided as an equation (9) from equlns T d 1,16 d,StrlbutlonSulscalc "^ed. The distribution 



ui = ti-ti' 



xu = lui/5 



(7) 
(8) 



Su=S(ti-xu) 2 /(5-1) (g) 



[0153] From this distribution, an evaluation value X is provided in accordance wtth an equation (10). 

X = 1 - Su (10) 
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the pressure sensor 101a, the processing goes to step ST6 to obtain the data of the non-contact time [time(i+1)\ p] 
until the pressure sensor 1 01 a is pressed. After the data of the non-contact time [time(i+1 )\ p] is obtained at step ST6, 
whether or not there is an input for ending is discriminated again at step ST2. 

[0163] At step ST7 as a result of discrimination to the effect that there is an input for ending at step ST2, the ratio of 
5 the contact time of the pressure sensor 101a and the non-contact time of the pressure sensor to the entire time period 
is calculated. That is, data of the dimensionless contact time and non-contact time is obtained. Specifically, the entire 
time period T of actual measurement is calculated in accordance with an equation (11), where time(i)' represents the 
actually measured time during which the pressure sensor 101a is pressed, and data ti' of the actually measured time 
as a dimensionless quantity is calculated in accordance with an equation (12). Thus, a set of data Di'[ti, p] of the actually 
10 measured contact pattern is calculated. 



T=Ztime(i)' (11) 



tr = time(l)'/T (12) 

[0164] At the next step STB, the evaluation value (evaluation signal) is calculated in accordance with the above- 
described procedure. Thus, the evaluation value can be obtained. On the basis of such an evaluation value, the user 

20 determination section 144 determines the user. 

[01 65] By thus using the stroking manner evaluation signal calculation section 1 43 and the user determination section 
144, the user identification unit 123 compares the user identification information (actually measured contact pattern) 
from the stroking manner detection section 142 with the registered user identification information (registered contact 
pattern) from the user information extractor 122 and identifies the user. The user identification unit 123 outputs the 

25 specified user (information) as a user label to the action schedule section 130, as described above. 

[01 66] The user recognition system in the robot device 1 is described above. By using the user identification system, 
the robot device 1 can identify the user and can react differently to different users. Thus, the entertainment property 
of the robot device 1 is improved. 

[0167] In the robot according to the present invention, on the basis of Information of a user detected by detection 
30 means for detecting information of a user, one user is identified from a plurality of identifiable users by identification 
means, and an action corresponding to the one user identified by the identification means is manifested by action 
control means. Therefore, the robot can identify one user from a plurality of identifiable users and can react corre- 
sponding to the one user. 

[0168] In the action deciding method for a robot according to the present invention, on the basis of information of a 
35 user detected by detection means, one user is identified from a plurality of identifiable users and an action corresponding 
to the identified one user is manifested. Therefore, the robot can identify one user from a plurality of identifiable users 
and can react corresponding to the one user. 



40 Claims 

1. A robot comprising: 

detection means for detecting information of a user; 
45 identification means for identifying one userfrom a plurality of identifiable users on the basis of the information 

of the user detected by the detection means; and 

action control means for manifesting an action corresponding to the one user identified by the identification 
means. 

50 2. The robot as claimed in claim 1 , wherein information about a plurality of users is registered in advance and held, 
and a plurality of pieces of action information are held corresponding to the plurality of users, 

the identification means identifies one user on the basis of the information of the user registered in advance 
and the information of the user detected by the detection means, and 
55 the action control means manifests an action on the basis of action information corresponding to the one user. 

3. The robot as claimed in claim 2, further comprising registration means for registering the information of the user 
in advance. 
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to the identified one user. °eiecnon means, and manifesting an action corresponding 



16 



EP 1 151 779 A2 




FIG.1 



17 



EP 1 151 779 A2 




18 



EP 1 151 779 A2 




CO 

d 

LL 



19 



EP1 151 779 A2 




oo 



CO 



20 



EP 1 151 779 A2 




21 



EP 1 151 779 A2 




22 



EP1 151 779 A2 




FIG.7 
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